{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/TavernAI/TavernAI/blob/main/colab/GPU.ipynb\" target=\"_blank\"><img src=\"https://www.tavernai.xyz/oic.png\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "hCpoIHxYcDGs"
      },
      "outputs": [],
      "source": [
        "# @title { run: \"auto\", form-width: \"110%\", display-mode: \"form\" }\n",
        "# Taken from KoboldAI colab. Improved by FunkEngine.\n",
        "%%html --isolated\n",
        "<audio autoplay=\"\" src=\"https://tavernai.xyz/silence.m4a\" loop>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d-Yihz3hAb2E"
      },
      "source": [
        "https://colab.research.google.com/github/TavernAI/TavernAI/blob/main/colab/GPU.ipynb<br>\n",
        "\n",
        "Works with:<br>\n",
        "KoboldAI https://github.com/KoboldAI/KoboldAI<br>\n",
        "<br>\n",
        "**Links**<br>\n",
        "TavernAI Github https://github.com/TavernAI/TavernAI<br>\n",
        "TavernAI Discord https://discord.gg/zmK2gmr45t<br>\n",
        "TavernAI Boosty https://boosty.to/tavernai<br>\n",
        "<pre>\n",
        " Tavern.AI/ \\ /  ^   ^ ^ ^    ~~~~ ^ \\     /  ^ ^   ^ ^/ ^  ^ \\/^  ^    \\\n",
        "         /^ ^\\ ^  ^ ^   ^ ^  ~~   ^   \\   /  ^  ^ ^   / ^ ^  ^/   ^ ^    \\\n",
        "        /^ ^ ^\\^   ^ ^ ^   _||____   ^ \\ /  ^  ^ ^   /       /  ^  ^  ^   \\\n",
        " /\\ /\\ /\\   ^  \\  /\\ /\\   /\\\\\\\\\\\\\\\\   ^ \\  ^ /\\ /\\ /\\   /\\ /\\ /\\  ^ ^  ^/\\\n",
        "//\\\\/\\\\/\\\\   ^  \\//\\\\/\\\\ /__\\\\\\\\\\\\\\\\  _, \\  //\\\\/\\\\/\\\\ //\\\\/\\\\/\\\\  ^ ^ //\\\\\n",
        "//\\\\/\\\\/\\\\       //\\\\/\\\\ |__|_|_|__|   \\__, //\\\\/\\\\/\\\\ //\\\\/\\\\/\\\\     ///\\\\\\\n",
        " || || (@＾◡＾)(≖ ‸ ≖*) ( ←_← )\\| /|   /\\ \\ヽ(°ㅂ°╬) |( Ψ▼ｰ▼)∈ (O_O; )  |||\n",
        "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~ ~~~~~ ~~~~~ ~~~~~  ~~~~~ ~~\n",
        "</pre>\n",
        "**TavernAI Launch Instructions** <br>\n",
        "1. Click the TavernAI launch button.<br>\n",
        "2. Sign in to Google Drive when asked. Your chats and cards are stored there. <br>\n",
        "3. After less than a minute links to your TavernAI session are spammed to output console. <br>\n",
        "4. Click anywhere on the wall of URLs... Hard to miss. TavernAI will open in a new tab. <br>\n",
        "   (All links are same.)<br>  \n",
        "5. Now if you're using NovelAI, OpenAI, Horde, Proxies, OobaBooga, You have an API back-end <br>\n",
        "   to give to TavernAI lined up already. **So have fun at the Tavern!** But remember to check back on this tab every 20-25 minutes. <br>\n",
        " <br> (Also you should switch the runtime type over to CPU only, since all it is doing is running TavernAI.) <br>\n",
        " <br>  \n",
        " <br>  \n",
        "**For those of you using the KoboldAI API back-end solution, you need to scroll down to the next cell, and...** <br>\n",
        "   Tell KoboldAI which specific AI model you want it to load for you... There are several popular options in the menu. <br>\n",
        "   **But you can pick any of the hundreds of models on huggingface, and copy/paste the \"AuthorName/ModelNameHere-xxB\" string in the KoboldAI cell.** <br>\n",
        "   Anything up tp and including a \"-13B\" will load and run here on Colab. \"-20B\" models, in very rare circumstances can just be squeezed in, but it barely fits and you OOM a lot.  \n",
        "   So hopefully now you understand that you can use whatever model you want to use, you do not need to \"ask\" for it to be \"added to the list\" you just clear out the field and paste the descriptor. <br>\n",
        "   So, lets review: <br>\n",
        " Steps to gt KoboldAI running a model of your choosing, once TavernAI is up and running are... <br>\n",
        "1. Go to the cell below this one, under your TavernAI links, once you have them and confirmed they load... <br>\n",
        "2. Choose a model from the list, or clear out the field(Without the list open!) and paste in the correct descriptor for the model you want.  <br>\n",
        "3. Launch the cell by hitting the play button to the left of it. Find something to do for at least 5 minutes while it prepares to load KoboldAI... <br>\n",
        "4. About 5 minutes after clicking the play button, you should see the lovely teals and magentas that are a sure sign that KoboldAI is beginning to load your model. All you have to do is Wait. (About another 5 minutes for a \"-7B\" or 8min for a \"-13B.\") Once your model is downloaded and streamed into the GPU... <br>\n",
        "5. Go to TavernAI tab you opened in step 4 of the previous section. -> open right top menu -> select \"Settings\" -> select KoboldAI api(usually it is selected by default) -> The API URL field in \"Settings\" is pre-set to \"127.0.0.1:5000/api\" **don't touch it.** <br>\n",
        "6. Click \"Connect\" button. It **WILL** connect... <br>\n",
        "7. For those of you who run your own **local** install of TavernAI and are explicitly here **just** for an API link.  This will not provide one that you can use.  Please see: https://github.com/TavernAI/TavernAI/blob/main/colab/API-ONLY_for_node.js_local-TavernAI_v0.98RC.ipynb for your copy+paste needs. <br>\n",
        "8. Once the ACTUAL link that specifically says \"API LINK:\" appears, you copy and paste that into the TavernAI install that runs on your computer. <br>\n",
        "9. At that point both TavernAI and KoboldAI will be connected together and ready to communicate. Check back in on the Colab tab here every so often in case you're being accused of being a robot. <br>\n",
        " <br>  \n",
        "   It really isn't that complicated, or difficult to do... <br>\n",
        " <br>  \n",
        "   If you're having issues beyond your ability to wrangle, hit us up in the TavernAI Discord server.  We'll sort you out. <br>\n",
        " <br>  \n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "hps3qtPLFNBb"
      },
      "outputs": [],
      "source": [
        "# @title # TavernAI { form-width: \"110%\", display-mode: \"form\" }\n",
        "# FunkEngine's implementation ©\n",
        "use_google_drive = True #@param {type:\"boolean\"}\n",
        "# @markdown #New option ↓↓↓↓↓↓HERE↓↓↓↓↓↓\n",
        "just_here_to_drink = False #@param {type:\"boolean\"}\n",
        "# @markdown ##If you're not running KoboldAI as your back-end API.\n",
        "# @markdown\n",
        "# @markdown (So you're using OpenAI, or NovelAI, or the Horde, or a Reverse-Proxy etc...)\n",
        "# @markdown\n",
        "# @markdown Tick the \"I'm just here to drink...\" option:\n",
        "# @markdown\n",
        "# @markdown And you'll get the TavernAI console output!\n",
        "# @markdown\n",
        "# @markdown Useful for debugging errors, watching your tokens come back...\n",
        "# @markdown\n",
        "# @markdown Keeping this tab (and your colab runtime!) alive in your browser too.\n",
        "import os\n",
        "import time\n",
        "import requests\n",
        "os.chdir('/')\n",
        "!rm -rf /content/sample_data/\n",
        "if use_google_drive:\n",
        "  from google.colab import drive\n",
        "  drive.mount('/content/drive', force_remount=True)\n",
        "  if not os.path.exists(\"/content/drive/MyDrive/TavernAI\"):\n",
        "    os.mkdir(\"/content/drive/MyDrive/TavernAI\")\n",
        "else:\n",
        "  if not os.path.exists(\"/content/drive\"):\n",
        "    os.mkdir(\"/content/drive\")\n",
        "  if not os.path.exists(\"/content/drive/MyDrive\"):\n",
        "    os.mkdir(\"/content/drive/MyDrive\")\n",
        "  if not os.path.exists(\"/content/drive/MyDrive/TavernAI\"):\n",
        "    os.mkdir(\"/content/drive/MyDrive/TavernAI\")\n",
        "\n",
        "os.chdir('/content/')\n",
        "if not os.path.isfile('/content/cloudflared-linux-amd64.deb'):\n",
        "    !!wget -q https://github.com/cloudflare/cloudflared/releases/latest/download/cloudflared-linux-amd64.deb\n",
        "    time.sleep(2)\n",
        "    os.chmod('/content/cloudflared-linux-amd64.deb', 0o777)\n",
        "    !!sudo dpkg -i cloudflared-linux-amd64.deb\n",
        "\n",
        "#TavernAI\n",
        "os.chdir('/')\n",
        "!node -v\n",
        "!npm -v\n",
        "!cloudflared -v\n",
        "\n",
        "if not os.path.exists(\"/content/TavernAI\"):\n",
        "  os.mkdir(\"/content/TavernAI\")\n",
        "  os.chdir('/content/TavernAI/')\n",
        "  !!git clone -b main https://github.com/TavernAI/TavernAI/ /content/TavernAI/\n",
        "  time.sleep(1)\n",
        "\n",
        "os.chdir('/content/TavernAI/')\n",
        "\n",
        "!!npm install\n",
        "time.sleep(1)\n",
        "\n",
        "if use_google_drive:\n",
        "  print(\"TavernAI 'Total User eXpericnce Persistence'\")\n",
        "  print(\"--courtesy of FunkEngine!\")\n",
        "  print(\"Starting with User Avatars...\")\n",
        "  os.chdir('/')\n",
        "  if not os.path.exists(\"/content/drive/MyDrive/TavernAI/User Avatars\"):\n",
        "    print(\"Setting up Google Drive to hold avatars.\")\n",
        "    !mv -v '/content/TavernAI/public/User Avatars' '/content/drive/MyDrive/TavernAI/'\n",
        "  os.chdir('/')\n",
        "  if os.path.exists(\"/content/drive/MyDrive/TavernAI/User Avatars\") and not os.path.islink(\"/content/TavernAI/public/User Avatars\"):\n",
        "    print(\"Using existing avatars from your Google Drive new avatars stored as well.\")\n",
        "    !rm -rf '/content/TavernAI/public/User Avatars'\n",
        "    !ln -fs '/content/drive/MyDrive/TavernAI/User Avatars/' '/content/TavernAI/public/'\n",
        "    os.chdir('/')\n",
        "    print(\"Avatars shall persist between sessions.\")\n",
        "  if not os.path.isfile(\"/content/drive/MyDrive/TavernAI/settings.json\"):\n",
        "    print(\"First time here? Copying default settings for you so they persist when you change them...\")\n",
        "    !cp -f -p \"/content/TavernAI/public/settings.json\" \"/content/drive/MyDrive/TavernAI/settings.json\"\n",
        "    os.chdir('/')\n",
        "  if os.path.isfile(\"/content/drive/MyDrive/TavernAI/settings.json\") and not os.path.islink(\"/content/TavernAI/public/settings.json\"):\n",
        "    print(\"Linking settings from your Google Drive to be used instead of defaults.\")\n",
        "    !rm -f \"/content/TavernAI/public/settings.json\"\n",
        "    !ln -fs \"/content/drive/MyDrive/TavernAI/settings.json\" \"/content/TavernAI/public/settings.json\"\n",
        "    os.chdir('/')\n",
        "  if not os.path.exists(\"/content/drive/MyDrive/TavernAI/characters\"):\n",
        "    print(\"Setting up Google Drive to hold characters.\")\n",
        "    !mv -v /content/TavernAI/public/characters /content/drive/MyDrive/TavernAI/\n",
        "    os.chdir('/')\n",
        "  if os.path.exists(\"/content/drive/MyDrive/TavernAI/characters\") and not os.path.islink(\"/content/TavernAI/public/characters\"):\n",
        "    print(\"Using existing characters from your Google Drive new characters will be stored there as well.\")\n",
        "    !rm -rf /content/TavernAI/public/characters\n",
        "    !ln -fs /content/drive/MyDrive/TavernAI/characters/ /content/TavernAI/public/\n",
        "    os.chdir('/')\n",
        "  if not os.path.exists(\"/content/drive/MyDrive/TavernAI/rooms\"):\n",
        "    print(\"More people more fun, right? Setting up Google Drive to also manage your rooms.\")\n",
        "    !mv -v /content/TavernAI/public/rooms /content/drive/MyDrive/TavernAI/\n",
        "    os.chdir('/')\n",
        "  if os.path.exists(\"/content/drive/MyDrive/TavernAI/rooms\") and not os.path.islink(\"/content/TavernAI/public/rooms\"):\n",
        "    print(\"Bringing in your multiplayer rooms from Google Drive. Try not to have too much fun!\")\n",
        "    !rm -rf /content/TavernAI/public/rooms\n",
        "    !ln -fs /content/drive/MyDrive/TavernAI/rooms/ /content/TavernAI/public/\n",
        "    os.chdir('/')\n",
        "  if not os.path.exists(\"/content/drive/MyDrive/TavernAI/chats\"):\n",
        "    print(\"Don't you rememeber what they took from us? Setting up Google Drive to remember the past.\")\n",
        "    !mv -v /content/TavernAI/public/chats /content/drive/MyDrive/TavernAI/\n",
        "    os.chdir('/')\n",
        "  if os.path.exists(\"/content/drive/MyDrive/TavernAI/chats\") and not os.path.islink(\"/content/TavernAI/public/chats\"):\n",
        "    print(\"Using your conversation history from Google Drive. Always remember what they took from us!\")\n",
        "    !rm -rf /content/TavernAI/public/chats\n",
        "    !ln -fs /content/drive/MyDrive/TavernAI/chats/ /content/TavernAI/public/\n",
        "    os.chdir('/')\n",
        "  if not os.path.exists(\"/content/drive/MyDrive/TavernAI/backgrounds\"):\n",
        "    print(\"I've heard talk that backgrounds are an extremely super important to some of you...\")\n",
        "    !mv -v /content/TavernAI/public/backgrounds /content/drive/MyDrive/TavernAI/\n",
        "    os.chdir('/')\n",
        "  if os.path.exists(\"/content/drive/MyDrive/TavernAI/backgrounds\") and not os.path.islink(\"/content/TavernAI/public/backgrounds\"):\n",
        "    print(\"------\")\n",
        "    !rm -rf /content/TavernAI/public/backgrounds\n",
        "    !ln -fs /content/drive/MyDrive/TavernAI/backgrounds/ /content/TavernAI/public/\n",
        "    print(\"Background storage linked!\")\n",
        "  print(\"That should be everything set up the way it was before...\")\n",
        "  print(\"TavernAI user experience continuity is implied but not guaranteed.\")\n",
        "elif not use_google_drive:\n",
        "  print(\"Ephemeral-Tavern engaged! (You chose not to use Drive.)\")\n",
        "  print(\"Your Chats... Your Cards... Your Settings...\")\n",
        "  print(\"Will __NOT__ be saved, Gone forever when this Colab runtime terminates.\")\n",
        "\n",
        "print(\"DONE!\")\n",
        "#LZ is CLEAR! We are GO for landing! (A seat at the bar in the tavern, of course!)\n",
        "os.chdir('/content/TavernAI/')\n",
        "if not just_here_to_drink:\n",
        "    !!nohup node server.js 2>&1 > /content/drive/MyDrive/TavernAI/console.log &\n",
        "    time.sleep(2)\n",
        "\n",
        "#Cloudflared travel size secure egress tunnel deploying...\n",
        "os.chdir('/content/')\n",
        "!!nohup cloudflared tunnel --url http://127.0.0.1:8000 2>&1 > cf.txt &\n",
        "time.sleep(4)\n",
        "scrape = open('cf.txt', 'r')\n",
        "haystack = scrape.read()\n",
        "scrape.close()\n",
        "scrape = ''\n",
        "lodestone = haystack.split(\"  https://\")[1]\n",
        "haystack = ''\n",
        "needle = lodestone.split('.trycloudflare.com')[0]\n",
        "lodestone = ''\n",
        "needle = (\"https://\" + needle + \".trycloudflare.com/\")\n",
        "#It's in here somewhere...\n",
        "#Something sharp and pointy turned up...\n",
        "os.chdir('/content/')\n",
        "if needle != '':\n",
        "  cloudflarelog = open('cloudflare.log', 'w')\n",
        "  cloudflarelog.write(needle)\n",
        "  cloudflarelog.close()\n",
        "print(\"TavernAI Link:\")\n",
        "print(needle)\n",
        "print(needle)\n",
        "print(needle)\n",
        "print(needle)\n",
        "print(needle)\n",
        "os.chdir('/content/TavernAI/')\n",
        "if just_here_to_drink:\n",
        "    !node server.js\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# @title <b>KoboldAI</b> { form-width: \"110%\", display-mode: \"form\" }\n",
        "Model = \"NeverSleep/Noromaid-13b-v0.1.1\" # @param [\"NeverSleep/Noromaid-13b-v0.1.1\", \"NeverSleep/Nethena-13B\", \"Undi95/Nete-13B\", \"KoboldAI/OPT-13B-Erebus\", \"Xwin-LM/Xwin-LM-13B-V0.2\", \"WizardLM/WizardLM-13B-V1.2\", \"baichuan-inc/Baichuan2-13B-Chat\", \"Undi95/MLewd-v2.4-13B\", \"codellama/CodeLlama-13b-Python-hf\", \"codellama/CodeLlama-13b-hf\", \"WizardLM/WizardCoder-15B-V1.0\", \"WizardLM/WizardCoder-Python-13B-V1.0\", \"Xwin-LM/XwinCoder-13B\", \"Xwin-LM/Xwin-Math-13B-V1.0\", \"WizardLM/WizardMath-13B-V1.0\", \"FunkEngine/SchweinZwei-13b\", \"FunkEngine/FunkEngine-6B\", \"Xwin-LM/Xwin-LM-7B-V0.1\", \"####↓↓WHY↓↓WOULD↓↓YOU↓↓STILL↓↓USE↓↓THESE? USE 13B on COLAB####\", \"KoboldAI/OPT-6B-nerys-v2\", \"KoboldAI/OPT-6.7B-Erebus\", \"KoboldAI/OPT-6.7B-Nerybus-Mix\", \"####2019↓↓CALLED↓↓THEY↓↓WANT↓↓THEIR↓↓SMOL↓↓MODEL↓↓BACK####\", \"KoboldAI/OPT-2.7B-Erebus\"] {allow-input: true}\n",
        "# FunkEngine's implementation© \n",
        "# KoboldAI by Henky!!© \n",
        "\n",
        "!!nvidia-smi -pm 1\n",
        "!!nvidia-smi -pl 69\n",
        "!!nvidia-smi -c 3\n",
        "!nvidia-smi\n",
        "import os\n",
        "os.chdir('/')\n",
        "!!nohup pip install typing >/dev/null\n",
        "!!nohup pip install --ignore-installed blinker==1.6.2\n",
        "!!rm -rf /content/sample_data/\n",
        "if not os.path.exists(\"/content/KoboldAI\"):\n",
        "  os.mkdir(\"/content/KoboldAI\")\n",
        "os.chdir('/content/KoboldAI/')\n",
        "!git clone -b united https://github.com/henk717/KoboldAI /content/KoboldAI/\n",
        "!!git submodule update --init --recursive\n",
        "os.chdir('/content/KoboldAI/')\n",
        "!!sudo apt install netbase aria2 -y\n",
        "!!pip install -r requirements.txt\n",
        "!nvidia-smi\n",
        "os.chdir('/content/KoboldAI/')\n",
        "!python3 aiserver.py --cacheonly --quiet --model $Model"
      ],
      "metadata": {
        "id": "kzzNdltKQsY3"
      },
      "execution_count": null,
      "outputs": []
    }
  ],
  "metadata": {
    "colab": {
      "provenance": [],
      "gpuType": "T4",
      "cell_execution_strategy": "setup",
      "include_colab_link": true,
      "private_outputs": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
